Evaluating Automatic Semantic Taggers

Unlike the problems of part-of-speech tagging and parsing, where commonly utilized training and test sets such as the Brown Corpus and Penn Treebank have existed for a number of years, evaluation of word sense disambiguation sytems is not yet standardized. In fact, most previous work in sense disambiguation has tended to use different sets of polysemous words, different sense inventories, different evaluation metrics and different test corpora. This working session will address these problems and seek solutions to them. Examples of issues for discussion include: